Papers with training procedure
Summary Level Training of Sentence Rewriting for Abstractive Summarization (D19-54)
Copied to clipboard
| Challenge: | Existing models rely on sentence-level rewards or suboptimal labels to achieve summary-level ROUGE scores. |
| Approach: | They propose a model that extracts salient sentences from a document and paraphrases them to generate a summary. |
| Outcome: | The proposed model improves on CNN/Daily Mail and New York Times datasets. |
Metaphorical Polysemy Detection: Conventional Metaphor Meets Word Sense Disambiguation (2022.coling-1)
Copied to clipboard
| Challenge: | Linguists distinguish between novel and conventional metaphors, a distinction which the metaphor detection task in NLP does not take into account. |
| Approach: | They propose a method which treats conventional metaphors as a property of word senses in a lexicon and combines metaphor detection with word sense disambiguation to train it. |
| Outcome: | The proposed model outperforms a state-of-the-art model in annotating metaphor in two subsets of WordNet and achieves .78 ROC-AUC score compared to baseline model . |
CCT-Code: Cross-Consistency Training for Multilingual Clone Detection and Code Search (2025.naacl-srw)
Copied to clipboard
Nikita Sorokin, Tikhonov Anton, Dmitry Abulkhanov, Ivan Sedykh, Irina Piontkovskaya, Valentin Malykh
| Challenge: | clone detection is crucial in software development for identifying semantically similar code . clones can be found in the same language code snippets, but there is little research on multilingual clonage detection. |
| Approach: | They propose a novel training procedure leveraging cross-lingual similarity to train language models on source code in various programming languages. |
| Outcome: | The proposed method achieves state-of-the-art on C++ and Python clone detection benchmarks with comparable performance on decoder-based models. |
A Personalized Dialogue Generator with Implicit User Persona Detection (2022.coling-1)
Copied to clipboard
| Challenge: | Existing models for personalized dialogue generation tend to be self-centered, with little care for the user in the dialogue. |
| Approach: | They propose a personalized dialogue generator by detecting an implicit user persona and using conditional variational inference to model the user's potential persona with no external knowledge. |
| Outcome: | The proposed model improves both automatic metrics and human evaluations by focusing on the user's persona and posterior-discriminated regularization. |
Multiple Character Embeddings for Chinese Word Segmentation (P19-2)
Copied to clipboard
| Challenge: | Chinese word segmentation is regarded as character-based sequence labeling task in most current work but it neglects important fact: Chinese characters contain both semantic and phonetic meanings. |
| Approach: | They propose a shared bi-LSTM-CRF model which fuses linguistic features efficiently by sharing the LSTM network during the training procedure. |
| Outcome: | The proposed model achieves state-of-the-art in AS and CityU corpora without external lexical resources. |
Diving Deep into Modes of Fact Hallucinations in Dialogue Systems (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Knowledge Graph(KG) grounded conversations often use large pre-trained models and suffer from fact hallucination. |
| Approach: | They propose to use a human feedback analysis to identify various modes of hallucination in KG chatbots. |
| Outcome: | The proposed system provides fine-grained signals that control fallacious content while generating responses. |
Unsupervised Labeled Parsing with Deep Inside-Outside Recursive Autoencoders (D19-1)
Copied to clipboard
| Challenge: | Existing models that use ground-truth part-of-speech tags are not always available and have significant weaknesses. |
| Approach: | They propose to use deep inside-outside recursive autoencoders to cluster the learned phrase vectors to induce span labels. |
| Outcome: | The proposed model outperforms ELMo and BERT on two versions of the Wall Street Journal dataset and improves over a previous state-of-the-art system that requires additional human annotations by 5 absolute F1 points (19% relative error reduction). |
Critic-Driven Decoding for Mitigating Hallucinations in Data-to-text Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Hallucination of text lacking grounding in input data is a problem in neural data-to-text generation. |
| Approach: | They propose to combine probabilistic output of a generator language model with the output of an “text critic” classifier which guides the generation by assessing the match between the input data and the generated text. |
| Outcome: | The proposed method improves on the WebNLG and OpenDialKG benchmarks. |
Multi-Granularity Representations of Dialog (D19-1)
Copied to clipboard
| Challenge: | Neural models of dialog rely on generalized latent representations of language. |
| Approach: | They propose a training procedure which explicitly learns multiple representations of language at several levels of granularity. |
| Outcome: | The proposed training procedure significantly improves performance on the next utterance retrieval task using the MultiWOZ dataset and the Ubuntu dialog corpus. |
Hierarchy Response Learning for Neural Conversation Generation (D19-1)
Copied to clipboard
| Challenge: | Neural conversation generation models can't perceive and express the intention effectively, causing dull and generic responses. |
| Approach: | They propose a hierarchical response generation model to capture conversation intention . they propose an expression reconstruction model and an expression attention model . |
| Outcome: | The proposed model can generate the responses with more appropriate content and expression. |
Extending Input Contexts of Language Models through Training on Segmented Sequences (2024.findings-naacl)
Copied to clipboard
| Challenge: | Effectively training languages models on long sequences poses many technical challenges. |
| Approach: | They propose a method for extending positional embeddings by sub-sampling segments from long inputs while maintaining their original position. |
| Outcome: | The proposed method extends the input con-text size of pretrained models without any changes in the model's memory and memory costs. |
Composing Structure-Aware Batches for Pairwise Sentence Classification (2022.findings-acl)
Copied to clipboard
| Challenge: | Identifying the relation between two sentences requires datasets with pairwise annotations. |
| Approach: | They propose three batch composition strategies to incorporate such information and measure their performance over 14 heterogeneous pairwise sentence classification tasks. |
| Outcome: | The proposed methods show that the pre-trained language model can benefit from having such structural information in a low-resource setting. |
Neural Decipherment via Minimum-Cost Flow: From Ugaritic to Linear B (P19-1)
Copied to clipboard
| Challenge: | Existing methods for decipherment of lost languages are limited by limited data and scarce quantities of ancient text. |
| Approach: | They propose a neural approach for automatic decipherment of lost languages . they use an expressive sequence-to-sequence model to capture character-level correspondences between cognates . |
| Outcome: | The proposed approach improves on the decipherment of Ugaritic and Linear B in ancient Greek . the proposed approach is highly customized for a given language pair and does not generalize to other lost languages. |
We Need to Talk About train-dev-test Splits (2021.emnlp-main)
Copied to clipboard
| Challenge: | Standard train-dev-test splits used to benchmark multiple models are now used in NLP . comparing multiple versions of the same model on the test data leads to overfitting and "expiration" of test sets. |
| Approach: | They propose to use a tune-set when developing neural network methods to do model picking. |
| Outcome: | The proposed model picker is more robust against the evaluated hyperparameter ranges than the standard split split. |
Continuation KD: Improved Knowledge Distillation through the Lens of Continuation Optimization (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for knowledge distillation (KD) do not mitigate the noise in the teacher’s output: modeling the noisy behaviour of the teacher can distract the student from learning more useful features. |
| Approach: | They propose a method that optimizes the highly non-convex KD objective by starting with the smoothed version of this objective and making it more complex as the training proceeds. |
| Outcome: | The proposed method achieves state-of-the-art performance on NLU and computer vision tasks. |
Learning to Retrieve Iteratively for In-Context Learning (2024.emnlp-main)
Copied to clipboard
Yunmo Chen, Tongfei Chen, Harsh Jhamtani, Patrick Xia, Richard Shin, Jason Eisner, Benjamin Van Durme
| Challenge: | In-context learning is a powerful tool for learning large language models. |
| Approach: | They propose an iterative retrieval framework that empowers retrievers to make iterable decisions through policy optimization. |
| Outcome: | The proposed framework outperforms existing methods on semantic parsing datasets with 4M additional parameters for state encoding. |
Certified Robustness to Adversarial Word Substitutions (D19-1)
Copied to clipboard
| Challenge: | State-of-the-art NLP models can be fooled by adversaries that apply seemingly innocuous label-preserving transformations to input text. |
| Approach: | They propose to train models that are provably robust to all word substitutions in a family of label-preserving transformations that can be replaced with a similar word without changing the original sentiment. |
| Outcome: | The proposed models achieve 75% adversarial accuracy on both sentiment analysis and natural language inference on IMDB and SNLI compared to models trained normally and ones trained with data augmentation. |
Plug and Play Autoencoders for Conditional Text Generation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Text autoencoders are used for conditional generation tasks such as style transfer. |
| Approach: | They propose a plug-and-play method where any pretrained autoencoder can be used and only requires learning a mapping within the embedding space. |
| Outcome: | The proposed method performs better than or comparable to strong baselines while being up to four times faster. |
Assessing the State of the Art in Scene Segmentation (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent advances in scene segmentation have made it difficult to detect scenes in literary texts. |
| Approach: | They propose to modify existing models to improve detection of scenes in literary texts . they propose to use a training sample generation scheme to alleviate this problem . |
| Outcome: | The proposed model is more robust to different types of texts, while its overall performance is slightly worse than that of BERT-based models. |
Learning Gender-Neutral Word Embeddings (D18-1)
Copied to clipboard
| Challenge: | Word embeddings trained on human-generated corpora inherit strong gender stereotypes . prior studies show such embeddables exhibit social biases, such as gender stereotype . |
| Approach: | They propose a method to preserve gender information in certain dimensions of word vectors . they propose GN-GloVe, which is a gender-neutral variant of the word embedding model . |
| Outcome: | The proposed method preserves gender information in certain dimensions of word vectors while compelling other dimensions to be free of gender influence. |
Birdie: Advancing State Space Language Modeling with Dynamic Mixtures of Training Objectives (2024.emnlp-main)
Copied to clipboard
| Challenge: | Efficient state space models struggle with tasks requiring in-context retrieval, such as text copying and associative recall, limiting their usefulness in practical settings. |
| Approach: | They propose a training procedure that improves the performance of SSMs on retrieval-intensive tasks such as phone book lookup, long paragraph question-answering, and infilling tasks. |
| Outcome: | The proposed training procedure improves performance on retrieval-intensive tasks that challenge current SSMs, such as phone book lookup, long paragraph question-answering, and infilling tasks. |
Multimodal Transformer Networks for End-to-End Video-Grounded Dialogue Systems (P19-1)
Copied to clipboard
| Challenge: | Existing work on video-grounded dialogue systems is limited by feature space and semantic information. |
| Approach: | They propose multimodal transformer networks to encode videos and incorporate information from different modalities. |
| Outcome: | The proposed system generates appropriate conversational response to queries of humans based on visual and audio aspects of a given video . it also generalizes to another multimodal visual-grounded dialogue task, and obtains promising performance. |
Spelling-Aware Construction of Macaronic Texts for Teaching Foreign-Language Vocabulary (D19-1)
Copied to clipboard
| Challenge: | a machine foreign-language teacher replaces word tokens with glosses in a foreign language to ease the human reader into understanding the L2 vocabulary. |
| Approach: | They propose a machine foreign-language teacher that modifies text by replacing word tokens with glosses in a foreign language to ease the human reader into understanding the L2 . |
| Outcome: | The proposed model can learn representations for novel words and is a proxy for word guessing and learning ability of real human students. |
Using Context in Neural Machine Translation Training Objectives (2020.acl-main)
Copied to clipboard
| Challenge: | Neural Machine Translation (NMT) training is based on document-level metrics, not sentence-level BLEU. |
| Approach: | They propose to merge document-level metrics with batch-level documents to improve NMT training. |
| Outcome: | The proposed training is more robust for document-level metrics than sequence MRT and maximum-likelihood training. |
Do Question Answering Modeling Improvements Hold Across Benchmarks? (2023.acl-long)
Copied to clipboard
| Challenge: | a new study finds that human-constructed and downsampled benchmarks hold more concurrence than downsampled benchmarks. |
| Approach: | They propose to measure concurrence between two QA benchmarks on a set of 20 models . they find that human-constructed benchmarks have high concurrence amongst themselves . |
| Outcome: | The proposed models hold broadly across the diverse landscape of question answering (QA) benchmarks. |
Privacy Implications of Retrieval-Based Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | a study of retrieval-based language models shows improved interpretability, factuality, and adaptability compared to parametric counterparts . kNN-LMs are more susceptible to leaking private information from their private datastore than parametric models . |
| Approach: | They present the first study of privacy risks in retrieval-based language models . they aim to strike a balance between utility and privacy in domains where privacy is of concern . |
| Outcome: | The proposed methods improve interpretability, factuality, and adaptability compared to parametric models . the study finds that kNN-LMs are more susceptible to leaking private data than parametric ones . |
Argument Relation Classification through Discourse Markers and Adversarial Training (2024.emnlp-main)
Copied to clipboard
| Challenge: | Argument relation classification (ARC) identifies supportive, contrasting and neutral relations between argumentative units. |
| Approach: | They propose an argument relation classifier that integrates knowledge of discourse markers into a pre-trained RoBERTa model. |
| Outcome: | The proposed model outperforms existing methods and learns discriminative sentence embeddings supporting the task. |
Controlling What You Share: Assessing Language Model Adherence to Privacy Preferences (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) are accessed via commercial APIs, but expose data to service providers. |
| Approach: | They propose a framework where a local model uses natural language instructions to rewrite queries and paired them with synthetic privacy profiles to achieve better privacy preservation. |
| Outcome: | The proposed model outperforms large-scale few-shot models in terms of privacy preservation and performance. |